Light flicker mitigation in machine vision systems

ABSTRACT

The present disclosure is directed to systems and methods of detecting and minimizing the effect of light flickering in a stitched HDR image formed using a plurality of images obtained using image acquisition sensor circuitry. Light flicker mitigation (LFM) circuitry receives a signal containing data representative of a plurality of images. Multi-feature extraction circuits in the LFM circuitry extract data representative of one or more features from some or all of the pixels included in the plurality of images. Light flicker detection and analysis circuits in the LFM circuitry detect flickering pixels in the plurality of images. Exposure index conversion circuits in the LFM circuitry determine one or more output corrections to apply to the plurality of images as the images are stitched to form a single HDR image in which flickering pixels are minimized, mitigated, or eliminated.

TECHNICAL FIELD

The present disclosure relates to machine vision systems, morespecifically to improving recognition and identification of switchedlight sources such as light emitting diode sources.

BACKGROUND

Light emitting diodes (LEDs) have found increasing use in motorvehicles, traffic control devices, traffic information devices,emergency vehicle signals, and motorway notifications due to theirrelatively low cost, long service life expectancy, low powerconsumption, relatively high efficiency and reliability, and simpleelectronic control. Thus, LED light flicker mitigation (LFM) has foundincreasing importance in Advanced Driver Assistance (ADAS) and CameraMonitoring Systems (CMS). In many instances, such ADAS and CMS systemsmake use of high-dynamic range (HDR) imaging to capture and reproduce awide range of colors and luminous intensities in various drivingsituations and lighting conditions to provide more realistic visualcontent. HDR technology is typically employed whenever scene or visualcontent exceeds the dynamic range of devices (commonly defined as aratio between the largest and smallest non-negative quantities an imageacquisition device is able to sense and/or reproduce). For example,current HDR image sensors generate an output that includes a number(e.g., three or four) images acquired using different exposure timesand/or gain values. A single output image is then generated when theindividual exposure images are combined, or stitched, by the imageacquisition sensor. In the alternative, multiple images, each havingdifferent exposure times and/or gain values, may be combined or stitchedusing one or more external or host devices, such as one or more imageand signal processors (ISPs).

With LED light sources, HDR stitching generally results in incompleteinformation content due to sensor saturation at longer exposure times orfailure to detect and avoid light flickering by an illuminated LED atshorter exposure times. An occurrence of either of these conditions mayadversely impact the operation of brake-light detection, night-visionassistance, and/or traffic-signal recognition systems. To avoid suchissues LFM systems may be employed.

BRIEF DESCRIPTION OF THE DRAWINGS

Features and advantages of various embodiments of the claimed subjectmatter will become apparent as the following Detailed Descriptionproceeds, and upon reference to the Drawings, wherein like numeralsdesignate like parts, and in which:

FIG. 1 is a schematic diagram of an illustrative system in which lightflicker mitigation (LFM) circuitry receives multi-exposure image dataproduced by image acquisition sensor circuitry, identifies those pixelsand/or pixel blocks demonstrating output variances indicative offlickering, and provides one or more signals containing correctioninformation to HDR stitching circuitry to produce an output image inwhich the flickering effects are reduced or eliminated, in accordancewith at least one embodiment described herein;

FIG. 2 depicts a block diagram of an illustrative system that includes3A (Auto-focus, Auto-exposure, Auto-white balance) circuitry that, usingthe output signal from the image acquisition sensor circuitry and anoutput signal from the LFM circuitry, generates an output signalcontaining exposure compensation data that is communicated to the imageacquisition sensor circuitry, in accordance with at least one embodimentdescribed herein;

FIG. 3 is a block diagram of a system that includes illustrative LFMcircuitry that includes: one or more multi-feature extraction circuits;one or more light flicker detection and analysis circuits; one or moreexposure index conversion circuits; and one or more adjustmentfactor/LFM statistic conversion circuits; in accordance with at leastone embodiment described herein;

FIG. 4 is a schematic diagram of an illustrative electronic,processor-based, device that includes a central processing unit (CPU) ormulti-chip module (MCM) that includes flicker detection and correctioncircuitry and/or graphical processing unit (GPU) circuitry that includeslight flicker detection and correction circuitry, in accordance with atleast one embodiment described herein;

FIG. 5 is a high-level flow diagram of an illustrative method ofdetecting and analyzing images for the presence of flickering pixels, inaccordance with at least one embodiment described herein;

FIG. 6 is a high-level flow diagram of an illustrative method ofadjusting or otherwise correcting one or more exposure settings and/orgain of the image acquisition sensor circuitry for each of the imagesincluded in the plurality of images acquired by the image acquisitionsensor circuitry, in accordance with at least one embodiment describedherein; and

FIG. 7 is a high-level flow diagram of an illustrative method ofcorrecting, altering, and/or adjusting the exposure settings and/or gainof one or more of the images collected or otherwise acquired by theimage acquisition sensor circuitry, in accordance with at least oneembodiment described herein.

Although the following Detailed Description will proceed with referencebeing made to illustrative embodiments, many alternatives, modificationsand variations thereof will be apparent to those skilled in the art.

DETAILED DESCRIPTION

The systems and methods described herein use signal processing methodsto restore captured LED information and/or content during the HDRstitching process. More specifically, the systems and methods describedherein use light flicker mitigation (LFM) circuitry to receivemulti-exposure image data from one or more image acquisition sensorscarried by a motor vehicle or used in other camera systems, such asvideo surveillance cameras, traffic cameras, etc. The LFM circuitrydetects the presence of pixels demonstrating a variable output over timethat is characteristic of flickering. The LFM circuitry enhances eitherindividual pixels and/or blocks of pixels identified as flickering tominimize the impact of the flickering on the resultant HDR stitchedimage. The LFM circuitry may also generate one or more exposure controloutputs that are communicated to the auto-focus, auto-exposure,auto-white balance (3A) circuitry that control image exposure by theimage acquisition sensor circuitry. In embodiments, the LFM circuitrymay include various modules, each containing dedicated circuitry capableof providing one or more functions to provide either or both HDRstitching correction and/or image acquisition sensor exposure adjustmentto produce images that mitigate the impact of flickering by adjustingthe exposure time and/or gain of each image acquired by the imageacquisition sensor and by identifying individual pixels or blocks ofpixels demonstrating flickering and adjusting the output parameters ofthe identified pixels during the HDR stitching process. Thus, thesystems and methods described herein beneficially and advantageouslyprovide correction information to the 3A circuitry controlling the imageacquisition sensor circuitry to mitigate the effects of flicker andprovide correction information to the HDR stitching circuitry directedtoward those pixels/pixel blocks identified as demonstrating flickering.

An image acquisition system is provided. The system may include: highdynamic range (HDR) sensor circuitry to obtain image data for each of aplurality of pixels included in an image; light flicker mitigation (LFM)circuitry to: receive the image data from the image acquisition sensor;perform multi-exposure feature extraction to identify pixels included inthe plurality of pixels that demonstrate flickering; determine, on aper-pixel basis, one or more correction factors to minimize, mitigate,or eliminate flickering of the identified pixels in a final HDR image;HDR stitching circuitry to: selectively combine pixel values using theimage data provided by the image acquisition sensor circuitry and thecorrection factors generated by the LFM circuitry to generate an outputHDR image.

An image acquisition method is provided. The method may include:obtaining, via high dynamic range (HDR) sensor circuitry, multi-exposureimage data for each of a plurality of pixels included in an image;receiving, by light flicker mitigation (LFM) circuitry, themulti-exposure image data from the image acquisition sensor circuitry;performing, by the LFM circuitry, multi-exposure feature extraction toidentify pixels included in the plurality of pixels that demonstrateflickering; determining, by the LFM circuitry on a per-pixel basis, oneor more correction factors to minimize, mitigate, reduce, avoid, oreliminate flickering of the identified pixels in a final HDR image; andselectively combine pixel values using the image data provided by theimage acquisition sensor circuitry and the correction factors generatedby the LFM circuitry to generate an output HDR image.

A non-transitory storage device is provided. The non-transitory storagedevice includes instructions that, when executed by light flickermitigation (LFM) circuitry, cause the LFM circuitry to: receive, fromhigh dynamic range (HDR) sensor circuitry, multi-exposure image data foreach of a plurality of pixels included in an image; performmulti-exposure feature extraction to identify pixels included in theplurality of pixels that demonstrate flickering; determine, on aper-pixel basis, one or more correction factors to minimize, mitigate,or eliminate flickering of the identified pixels in a final HDR image;and cause HDR stitching circuitry to selectively combine pixel valuesusing the image data provided by the image acquisition sensor circuitryand the correction factors generated by the LFM circuitry to generate anoutput HDR image.

An image acquisition system is provided. The system may include: meansfor obtaining, multi-exposure image data for each of a plurality ofpixels included in an image; means for receiving the multi-exposureimage data from the image acquisition sensor circuitry; means forperforming multi-exposure feature extraction to identify pixels includedin the plurality of pixels that demonstrate flickering; means fordetermining, on a per-pixel basis, one or more correction factors tominimize, mitigate, reduce, avoid, or eliminate flickering of theidentified pixels in a final HDR image; and means for selectivelycombine pixel values using the image data provided by the imageacquisition sensor circuitry and the correction factors generated by theLFM circuitry to generate an HDR image.

FIG. 1 is a schematic diagram of an illustrative system 100 in whichlight flicker mitigation (LFM) circuitry 120 receives multi-exposureimage data 112 generated, formed, or otherwise produced by imageacquisition sensor circuitry 110, identifies those pixels and/or pixelblocks demonstrating output variances indicative of flickering, andprovides one or more signals 122 containing correction information toHDR stitching circuitry 130 to produce an output image 132 in which theflickering effects are minimized, mitigated, reduced, avoided, oreliminated, in accordance with at least one embodiment described herein.As depicted in FIG. 1, in embodiments, image acquisition sensorcircuitry 110 gathers information and/or data associated with image 102containing objects within the field of view of the image acquisitionsensor circuitry 110. In embodiments, the image acquisition sensorcircuitry 110 may be operably coupled to a motor vehicle and acquiredimages 102 may include objects such as LED light sources such as trafficcontrol devices, motor vehicle indicators, and similar. The imageacquisition sensor circuitry 110 generates an output signal 112 thatincludes information and/or data representative of the scene with LEDlight sources within the field-of-view of the HDR image sensor 110. TheLFM circuitry 120 and the HDR stitching circuitry 130 receive the HDRimage sensor output signal 112. The LFM circuitry 120 identifies pixelsand/or pixel blocks demonstrating output parameters or other behaviorsindicative of flickering and determines one or more correction and/oradjustment parameters to apply in the HDR stitching process. The LFMcircuitry 120 generates an output signal 132 that includes informationand/or data representative of the one or more correction and/oradjustment parameters to apply during the HDR stitching process andcommunicates the output signal to the HDR stitching circuitry 130.

The image acquisition sensor circuitry 110 includes any number and/orcombination of electronic components, semiconductor devices, and/orlogic elements capable of acquiring a plurality of images(multi-exposure image data) of objects disposed or positioned within thefield-of-view 102 of the image acquisition sensor circuitry 110. Theimage acquisition sensor circuitry 110 produces an output signal 112that includes information and/or data representative of each of theplurality of images acquired by the image acquisition sensor circuitry110. The image data includes information and/or data associated witheach of a plurality of pixels that forms each respective one of theimages. Since each of the images is usually acquired using differentexposure settings and/or different gain settings, the same pixel orblock of pixels may have different characteristics (luminancecomponents, pixel values, etc.) from one image to another image. It isthis difference between images that enables the detection and/oridentification of flickering pixels and/or pixel blocks by the LFMcircuitry 120.

The LFM circuitry 120 includes any number and/or combination ofcurrently available and/or future developed electronic components,semiconductor devices, and/or logic elements capable of receivingmulti-exposure image data from the image acquisition sensor circuitry110, identifying pixels and/or pixel blocks demonstrating behaviorindicative of flickering, and determining one or more correction and/oradjustment parameters to apply to the identified flickering pixelsduring the HDR stitching process to minimize, mitigate, reduce, avoid,or eliminate the flickering effect in the final HDR image.

The HDR stitching circuitry 130 includes any number and/or combinationof currently available and/or future developed electronic components,semiconductor devices, and/or logic elements capable of receiving theoutput signal 112 containing information and/or data representative ofmultiple images from the image acquisition sensor circuitry 110 and anoutput signal 122 that includes correction and/or adjustment parameterinformation and/or data associated with pixels and/or pixel groupsincluded in the images that contain information indicative of flickeringpixels and/or pixel groups, and combining the images to produce anoutput signal in which the pixels and/or pixel blocks identified by theLFM circuitry 120 as demonstrating flickering have been adjusted toreduce, minimize, mitigate, avoid, or eliminate the flickering effect inthe output HDR image.

FIG. 2 depicts a block diagram of an illustrative system 200 thatincludes 3A (Auto-focus, Auto-exposure, Auto-white balance) circuitry210 that, using the output signal 112 from the image acquisition sensorcircuitry 110 and an output signal 202 from the LFM circuitry 120,generates an output signal containing exposure compensation data that iscommunicated to the image acquisition sensor circuitry 110, inaccordance with at least one embodiment described herein. Inembodiments, the LFM circuitry 120 may include one or more circuits todetermine one or more exposure parameters useful for mitigating,minimizing, or eliminating the flickering effects of one or more pixelsand/or pixels groups included in the image data in the output signal 112provided by the image acquisition sensor circuitry 110. The LFMcircuitry 120 generates an output signal 202 that includes informationand/or data representative of the corrected exposure parameters that iscommunicated to the 3A circuitry 210. The 3A circuitry 210 receives thecorrected exposure parameters and directly or indirectly causes theimage acquisition sensor circuitry 110 to implement the correctedexposure parameters.

FIG. 3 is a block diagram of a system 300 that includes illustrative LFMcircuitry 120 that includes: one or more multi-feature extractioncircuits 310; one or more flicker detection and analysis circuits 320;one or more exposure index conversion circuits 330; and one or moreadjustment factor/LFM statistic conversion circuits 340; in accordancewith at least one embodiment described herein. As depicted in FIG. 3,the LFM circuitry 120 receives the output signal 112 containing themulti-exposure image data generated by the image acquisition sensorcircuitry 110. The one or more multi-feature extraction circuits 310extract values from the pixels included in some or all of the pluralityof images provided by the image acquisition sensor circuitry 110. Thevalues extracted by the one or more multi-feature extraction circuits310 may use actual pixel values and/or one or more luminance componentsassociated with each of the pixels and/or pixel blocks.

The one or more multi-feature extraction circuits 310 include any numberand/or combination of currently available and/or future developedelectronic components, semiconductor devices, and/or logic elementscapable of extracting, from at least a portion of the image dataprovided by the image acquisition sensor circuitry 110, informationand/or data (e.g., features) useful for performing flicker detection andanalysis. In embodiments, the one or more multi-feature extractioncircuits 310 may include one or more programmable logic elements capableof executing machine-executable instruction sets that enable theextraction of information and/or data (e.g., features) useful forperforming flicker detection and analysis on the multi-exposure imagedata provided by the image acquisition sensor circuitry 110.

The multi-feature extraction circuit 310 performs multi-exposure featureextraction using actual pixel values or their luminance components ofpixels and/or pixel blocks contained in the multi-exposure image datareceived from the image acquisition sensor circuitry 110. Inembodiments, when the image acquisition sensor circuitry 110 provides aplurality of exposure images in Bayer color filter array (CFA) format,the multi-feature extraction circuit 310 may determine a valuecorresponding to pixel/pixel block luminance via Gaussian filtering witha 3×3 spatial window with coefficients [1 2 1; 2 4 2; 1 2 1]/16. Inembodiments, the multi-feature extraction circuit 310 may split each HDRimage into overlapping or non-overlapping (e.g., 2×2 Bayer quads) blocksand the luminance component can be approximated as the maximum, mean, orminimum pixel value in each such block. In embodiments, themulti-feature extraction circuit 310 beneficially uses the maximumoperator to leverage the favorable signal-to-noise ratios anddetectability of image areas affected by sensor saturation. In case ofthree-channel color images, such as RGB images obtained from Bayerimages via demosaicing, the multi-feature extraction circuit 310 mayobtain the luminance component in each pixel location as linearcombination (weighted average, e.g., (1R+2G+1B)/4 or0.299R+0.587G+0.114B) of red (R), green (G), and blue (B) components.

The one or more flicker detection and analysis circuits 320 include anynumber and/or combination of currently available and/or future developedelectronic components, semiconductor devices, and/or logic elementscapable of comparing pixels values and/or luminance components from thesame pixel and/or pixel blocks across multiple images to detectflickering pixels and/or pixel blocks. In embodiments, the one or moreflicker detection and analysis circuits 320 may include one or moreprogrammable logic elements capable of executing machine-executableinstruction sets that enable the comparison of pixels values, luminancecomponents, and/or other suitable representative values from the samepixel and/or pixel blocks across multiple images to detect flickeringpixels and/or pixel blocks. In embodiments, the one or more flickerdetection and analysis circuits 320 may perform hard thresholding (e.g.,binary output); soft thresholding (e.g., probability-like output); orsimilar measurement technique (e.g., exponential function in case ofpixel values or luminance differences). In embodiments, the one or moreflicker detection and analysis circuits 320 may receive definedthreshold values or some other control parameters as an input or mayadaptively determine such threshold values or other parameters based onthe content of the image data received from the HDR image sensors 110.

In embodiments, the one or more flicker detection and analysis circuits320 may compare a first image with a second image (a long exposure imageis compared to a medium exposure image, a medium exposure image iscompared to a short exposure image, etc.) in the acquired multi-exposureimage data or may compare each image included in a subset of imagesselected from the plurality of images. In embodiments, the one or moreflicker detection and analysis circuits 320 may determine pixel orluminance differences across the multi-exposure image capture usingexposure images compensated (normalized) for differences in exposuretime. In embodiments, the one or more flicker detection and analysiscircuits 320 may determine differences between a first (original)component and a second component adjusted using scale and offsetfactors. In embodiments, the one or more flicker detection and analysiscircuits 320 may determine one or more pixel value ratios or luminancecomponent ratios across some or all of the multi-exposure images. Inembodiments, the one or more flicker detection and analysis circuits 320may evaluate multi-exposure features in pixel locations or image regionsaffected by sensor saturation.

The one or more exposure index conversion circuits 330 include anynumber and/or combination of currently available and/or future developedelectronic components, semiconductor devices, and/or logic elementscapable of converting the output generated by the one or more flickerdetection and analysis circuits 320 with the identified flickeringpixels and/or pixel blocks to one or more values representative of anexposure index that includes data representative of the contribution ofthe pixels and/or pixel blocks in some or all of the individual exposureimages to the final corrected pixel value in the stitched HDR image 132produced by the HDR stitching circuitry 130. In embodiments, the one ormore exposure index conversion circuits 330 may include one or moreprogrammable logic elements capable of converting the output with theidentified flickering pixels and/or pixel blocks to one or more valuesrepresentative of an exposure index that includes data representative ofthe contribution of the pixels and/or pixel blocks in some or all of theindividual images to the final corrected pixel value in the stitched HDRimage 132 produced by the HDR stitching circuitry 130.

In embodiments, the one or more exposure index conversion circuits 330may convert the output by assigning individual exposure images to somevalues in the output detection range (e.g., for the pair of long andmedium exposures, the minimum value in the detection range can representthe long exposure, whereas the maximum value can represent the mediumexposure) and then determining the actual exposure index as a selection(for binary decision) or weighted average of these initial values orindices. In embodiments, the one or more exposure index conversioncircuits 330 may repeat this process for different pairs of exposureimages and intermediate results combined to obtain the final exposureindex, which is then provided to the HDR stitching circuitry 130 (alsotermed as HDR merge, HDR formation, or DOL processing). In embodiments,this process can be applied per pixel location, block, image region,etc. In embodiments, the obtained exposure indices and/or extractedfeatures may be subject to enhancement via low-pass filtering (e.g.,mean, Gaussian, or median filtering) and/or morphological processing(e.g., dilation, erosion, opening, closing) using, respectively, theindices and/or extracted features in their neighborhood to deal with thenoise present in the exposure images or possible processinginaccuracies. In embodiments, a plurality of input captured images, asubset of these images, luminance versions of these images, or someother suitable representative versions of these images can also besubject to low-pass filtering, such mean, weighted-average, anddetail-preserving (e.g., bilateral and non-local mean) filtering.

The one or more adjustment factor/LFM statistic conversion circuits 340include any number and/or combination of currently available and/orfuture developed electronic components, semiconductor devices, and/orlogic elements capable of collecting, gathering, or otherwise obtainingimage data from at least some of the plurality of images in the signal112 received from the image acquisition sensor circuitry 110. Inembodiments, the one or more exposure index conversion circuits 330 mayinclude one or more programmable logic elements capable of collecting,gathering, or otherwise obtaining image data from at least some of theplurality of images in the signal 112 received from the imageacquisition sensor circuitry 110.

In embodiments, the one or more adjustment factor/LFM statisticconversion circuits 340 use at least a portion of the information and/ordata extracted from the multi-exposure image capture to generate one ormore output signals 202 used to control the auto-exposure (AE) processperformed by the 3A circuitry 210. For example, in some embodiments, theone or more adjustment factor/LFM statistic conversion circuits 340convert information and/or data extracted from one or more of theplurality of images to additive or multiplicative factors to adjust theexposure times determined by the 3A circuitry 210 for each of at leastsome of the plurality of images. In embodiments, the one or moreadjustment factor/LFM statistic conversion circuits 340 calculate one ormore representative values for each of at least some of the plurality ofimages (e.g., the average, weighted average, median, luminance, ormaximum value from all detected pixels affected by light flickering),comparing these values to one or more predetermined thresholds (e.g.,sensor saturation value or its scaled version, such as 95% of themaximum value the image acquisition sensor circuitry 110 is able toread), and converting the difference or the ratio of these two values toactual exposure adjustment factors, which are then provided as an outputsignal 202 to the 3A circuitry 210. In embodiments, the LFM circuitry120 may generate a tensor or array containing image exposure statistics,such as mean values per block of predetermined dimensions. Inembodiments, such statistics tensors or arrays may be subject toenhancement via low-pass filtering (e.g., mean, Gaussian, or medianfiltering) and/or morphological processing (e.g., dilation, erosion,opening, closing) to deal with the noise present in the exposure imagesor possible processing inaccuracies. The LFM circuitry 120 communicatesthe tensor or array to the 3A circuitry 210, thereby allowing the 3Acircuitry to calculate one or more exposure adjustment factors or todetermine more optimal exposure times for some or all of the pluralityof images.

FIG. 4 is a schematic diagram of an illustrative electronic,processor-based, device 400 that includes a central processing unit(CPU) or multi-chip module (MCM) 410 that includes flicker detection andcorrection circuitry 100/200 and/or graphical processing unit (GPU)circuitry 412 that includes flicker detection and correction circuitry100/200, in accordance with at least one embodiment described herein.The processor-based device 400 includes image acquisition sensorcircuitry 110 capable of generating an output signal 112 that includesinformation and/or data representative of a plurality of images, eachobtained using different exposure settings and/or gain, that containinformation and/or data associated with object appearing within thefield-of-view 102 of the image acquisition sensor circuitry 110. Theprocessor-based device 400 may additionally include one or more of thefollowing: a wireless input/output (I/O) interface 420, a wired I/Ointerface 430, system memory 440, power management circuitry 450, anon-transitory storage device 460, and a network interface 170 used tocommunicatively couple the processor-based device 400 to one or moreexternal devices (e.g., a cloud-based server) 490 via one or morenetworks 480. The following discussion provides a brief, generaldescription of the components forming the illustrative processor-baseddevice 400. Example, non-limiting processor-based devices 400 mayinclude, but are not limited to: autonomous motor vehicles,semi-autonomous motor vehicles, manually controlled motor vehicles,smartphones, wearable computers, portable computing devices, handheldcomputing devices, desktop computing devices, blade server devices,workstations, and similar.

The processor-based device 400 includes processor circuitry 410 that mayprovide all or a portion of the LFM circuitry 120, the HDR stitchingcircuitry 130, and/or the 3A circuitry 210. In embodiments, some or allof the LFM circuitry 120, the HDR stitching circuitry 130, and/or the 3Acircuitry 210 may be in the form of an application specific integratedcircuit (ASIC) or similar chiplet-based integrated circuit that isdisposed along with the processor circuitry 410 as a system-on-chip(SoC) or Multi-Chip Module (MCM) semiconductor package. In embodiments,the processor-based device 400 includes GPU circuitry 412 that mayprovide all or a portion of the LFM circuitry 120, the HDR stitchingcircuitry 130, and/or the 3A circuitry 210. In embodiments, some or allof the LFM circuitry 120, the HDR stitching circuitry 130, and/or the 3Acircuitry 210 may be in the form of an application specific integratedcircuit (ASIC) or similar chiplet-based integrated circuit that isdisposed along with the GPU circuitry 412 as a system-on-chip (SoC) orMulti-Chip Module (MCM) semiconductor package.

Those skilled in the relevant art will appreciate that the illustratedembodiments as well as other embodiments may be practiced with otherprocessor-based device configurations, including portable electronic orhandheld electronic devices, for instance smartphones, portablecomputers, wearable computers, consumer electronics, personal computers(“PCs”), network PCs, minicomputers, server blades, mainframe computers,and the like. The processor circuitry 410 may include any number ofhardwired or configurable circuits, some or all of which may includeprogrammable and/or configurable combinations of electronic components,semiconductor devices, and/or logic elements that are disposed partiallyor wholly in a PC, server, or other computing system capable ofexecuting machine-readable instructions.

The processor-based device 400 includes a bus or similar communicationslink 416 that communicably couples and facilitates the exchange ofinformation and/or data between various system components including theprocessor circuitry 410, the graphics processor circuitry 412, one ormore wireless I/O interfaces 420, one or more wired I/O interfaces 430,the system memory 440, one or more storage devices 460, and/or thenetwork interface circuitry 470. The processor-based device 400 may bereferred to in the singular herein, but this is not intended to limitthe embodiments to a single processor-based device 400, since in certainembodiments, there may be more than one processor-based device 400 thatincorporates, includes, or contains any number of communicably coupled,collocated, or remote networked circuits or devices.

The processor circuitry 410 may include any number, type, or combinationof currently available or future developed devices capable of executingmachine-readable instruction sets. The processor circuitry 410 mayinclude but is not limited to any current or future developed single- ormulti-core processor or microprocessor, such as: on or more systems on achip (SOCs); central processing units (CPUs); digital signal processors(DSPs); graphics processing units (GPUs); application-specificintegrated circuits (ASICs), programmable logic units, fieldprogrammable gate arrays (FPGAs), and the like. Unless describedotherwise, the construction and operation of the various blocks shown inFIG. 4 are of conventional design. Consequently, such blocks need not bedescribed in further detail herein, as they will be understood by thoseskilled in the relevant art. The bus 416 that interconnects at leastsome of the components of the processor-based device 400 may employ anycurrently available or future developed serial or parallel busstructures or architectures.

The system memory 440 may include read-only memory (“ROM”) 442 andrandom access memory (“RAM”) 446. A portion of the ROM 442 may be usedto store or otherwise retain a basic input/output system (“BIOS”) 444.The BIOS 444 provides basic functionality to the processor-based device400, for example by causing the processor circuitry 410 to load and/orexecute one or more machine-readable instruction sets 414. Inembodiments, at least some of the one or more machine-readableinstruction sets 414 cause at least a portion of the processor circuitry410 to provide, create, produce, transition, and/or function as adedicated, specific, and particular machine, for example a light flickerdetection and analysis system that includes some or all of the LFMcircuitry 120, the HDR stitching circuitry 130, and/or the 3A circuitry210.

The processor-based device 400 may include at least one wirelessinput/output (I/O) interface 420. The at least one wireless I/Ointerface 420 may be communicably coupled to one or more physical outputdevices 422 (tactile devices, video displays, audio output devices,hardcopy output devices, etc.). The at least one wireless I/O interface420 may communicably couple to one or more physical input devices 424(pointing devices, touchscreens, keyboards, tactile devices, etc.). Theat least one wireless I/O interface 420 may include any currentlyavailable or future developed wireless I/O interface. Example wirelessI/O interfaces include, but are not limited to: BLUETOOTH®, near fieldcommunication (NFC), and similar.

The processor-based device 400 may include one or more wiredinput/output (I/O) interfaces 430. The at least one wired I/O interface430 may be communicably coupled to one or more physical output devices422 (tactile devices, video displays, audio output devices, hardcopyoutput devices, etc.). The at least one wired I/O interface 430 may becommunicably coupled to one or more physical input devices 424 (pointingdevices, touchscreens, keyboards, tactile devices, etc.). The wired I/Ointerface 430 may include any currently available or future developedI/O interface. Example wired I/O interfaces include but are not limitedto: universal serial bus (USB), IEEE 1394 (“FireWire”), and similar.

The processor-based device 400 may include one or more communicablycoupled, non-transitory, data storage devices 460. The data storagedevices 460 may include one or more hard disk drives (HDDs) and/or oneor more solid-state storage devices (SSDs). The one or more data storagedevices 460 may include any current or future developed storageappliances, network storage devices, and/or systems. Non-limitingexamples of such data storage devices 460 may include, but are notlimited to, any current or future developed non-transitory storageappliances or devices, such as one or more magnetic storage devices, oneor more optical storage devices, one or more electro-resistive storagedevices, one or more molecular storage devices, one or more quantumstorage devices, or various combinations thereof. In someimplementations, the one or more data storage devices 460 may includeone or more removable storage devices, such as one or more flash drives,flash memories, flash storage units, or similar appliances or devicescapable of communicable coupling to and decoupling from theprocessor-based device 400.

The one or more data storage devices 460 may include interfaces orcontrollers (not shown) communicatively coupling the respective storagedevice or system to the bus 416. The one or more data storage devices460 may store, retain, or otherwise contain machine-readable instructionsets, data structures, program modules, data stores, databases, logicalstructures, and/or other data useful to the processor circuitry 410and/or graphics processor circuitry 412 and/or one or more applicationsexecuted on or by the processor circuitry 410 and/or graphics processorcircuitry 412. In some instances, one or more data storage devices 460may be communicably coupled to the processor circuitry 410, for examplevia the bus 416 or via one or more wired communications interfaces 430(e.g., Universal Serial Bus or USB); one or more wireless communicationsinterfaces 420 (e.g., Bluetooth®, Near Field Communication or NFC);and/or one or more network interfaces 470 (IEEE 802.3 or Ethernet, IEEE802.11, or WiFi®, etc.).

Machine-readable instruction sets 414 and other programs, applications,logic sets, and/or modules may be stored in whole or in part in thesystem memory 440. Such instruction sets 414 may be transferred, inwhole or in part, from the one or more data storage devices 460. Theinstruction sets 414 may be loaded, stored, or otherwise retained insystem memory 440, in whole or in part, during execution by theprocessor circuitry 410 and/or graphics processor circuitry 412.

The processor-based device 400 may include power management circuitry450 that controls one or more operational aspects of the energy storagedevice 452. In embodiments, the energy storage device 452 may includeone or more primary (i.e., non-rechargeable) or secondary (i.e.,rechargeable) batteries or similar energy storage devices. Inembodiments, the energy storage device 452 may include one or moresupercapacitors or ultracapacitors. In embodiments, the power managementcircuitry 450 may alter, adjust, or control the flow of energy from anexternal power source 454 to the energy storage device 452 and/or to theprocessor-based device 400. The power source 454 may include, but is notlimited to, a solar power system, a commercial electric grid, a portablegenerator, an external energy storage device, or any combinationthereof.

For convenience, the processor circuitry 410, the graphics processorcircuitry 412, the wireless I/O interface 420, the wired I/O interface430, the system memory 440, the power management circuitry 450, thestorage device 460, and the network interface 470 are illustrated ascommunicatively coupled to each other via the bus 416, thereby providingconnectivity between the above-described components. In alternativeembodiments, the above-described components may be communicativelycoupled in a different manner than illustrated in FIG. 4. For example,one or more of the above-described components may be directly coupled toother components, or may be coupled to each other, via one or moreintermediary components (not shown). In another example, one or more ofthe above-described components may be integrated into the processorcircuitry 410 and/or the graphics processor circuitry 412. In someembodiments, all or a portion of the bus 416 may be omitted and thecomponents are coupled directly to each other using suitable wired orwireless connections.

FIG. 5 is a high-level flow diagram of an illustrative method 500 ofdetecting and analyzing images for the presence of flickering pixels, inaccordance with at least one embodiment described herein. Inembodiments, the HDR image sensor circuitry 110 obtains a plurality ofimages, each acquired using its own capture settings, such as theexposure time, analog and digital gains, etc. Various objects, such asswitched light sources (e.g., LED light sources), may provide theappearance of flickering in the acquired images. Such flickering maycause safety or operational concerns when motor vehicle informationaland/or control systems are unable to properly detect or analyze thecontent of the acquired images. The method 500 minimizes, mitigates, oreven eliminates the occurrence of flickering in the final HDR stitchedimage produced by the HDR stitching circuitry 130 and commences at 502.

At 504, the image acquisition sensor circuitry 110 generates one or moreoutput signals 112 that include information and/or data representativeof each of a plurality of images, each composed of a number of pixelsand/or pixel blocks and each obtained using its own capture settings,such as the exposure time, analog and digital gains, etc. Each of theimages included in the multi-exposure image capture contains informationand/or data representative of one or more objects, such as one or moreLED light sources, disposed in the field-of-view 102 of the imageacquisition sensor circuitry 110.

At 506, the LFM circuitry 120 receives the signal 112 that includes theinformation and/or data representative of at least a portion of theplurality of images obtained or otherwise acquired by the imageacquisition sensor circuitry 110.

At 508, the LFM circuitry 120 performs multi-exposure feature extractionusing the received data representative of the plurality of images. Inembodiments, the LFM circuitry 120 extracts, from at least a portion ofthe image data provided by the image acquisition sensor circuitry 110,information and/or data (e.g., features) useful for performing flickerdetection and analysis.

At 510, the LFM circuitry 120 determines, calculates, or otherwiseobtains one or more correction parameters and/or adjustment parametersto adjust values associated with pixels and/or pixel groups identifiedby the LFM circuitry 120 as flickering at 508. The LFM circuitry 120generates an output signal 122 that includes information and/or datarepresentative of the determined correction parameters and/or adjustmentparameters to minimize, mitigate, alter, avoid, or eliminate the effectsof flickering pixels and/or pixel blocks in the HDR image generated bythe HDR stitching circuitry 130.

At 512, the HDR stitching circuitry 130 receives the output signal 122from the LFM circuitry 120 and uses the correction factors included inthe signal to generate a single HDR output image 132 using theinformation and/or data representative of some or all of the imagesincluded in the image data generated by the image acquisition sensorcircuitry 110. The method 500 concludes at 514.

FIG. 6 is a high-level flow diagram of an illustrative method 600 ofadjusting or otherwise correcting one or more capture settings (e.g.,exposure time, analog and digital gains) of the image acquisition sensorcircuitry 110 for each of the images included in the multi-exposureimage data acquired by the image acquisition sensor circuitry 110, inaccordance with at least one embodiment described herein. Inembodiments, the LFM circuitry 120 analyzes the multi-exposure imagedata and generates an output signal 202 that includes at least one of:exposure correction data and gain correction data. The LFM circuitry 120generates an output signal 202 that includes the corrected exposureand/or gain settings or values and communicates the output signal 202 tothe 3A circuitry 210. The method 600 commences at 602.

At 604, the image acquisition sensor circuitry 110 generates one or moreoutput signals 112 that include information and/or data representativeof each of a plurality of images, each composed of a number of pixelsand/or pixel blocks and each obtained its own capture settings, such asthe exposure time, analog and digital gains, etc. Each of the imagesincluded in the plurality of images contains information and/or datarepresentative of one or more objects, such as one or more LED lightsources, disposed in the field-of-view 102 of the image acquisitionsensor circuitry 110.

At 606, the LFM circuitry 120 receives the signal 112 that includes theinformation and/or data representative of at least a portion of theplurality of images obtained or otherwise acquired by the imageacquisition sensor circuitry 110.

At 608, the LFM circuitry 120 performs multi-exposure feature extractionusing the received data representative of the plurality of images. Inembodiments, the LFM circuitry 120 extracts, from at least a portion ofthe image data provided by the image acquisition sensor circuitry 110,information and/or data (e.g., features) useful for performing flickerdetection and analysis.

At 610, the LFM circuitry 120 determines, calculates, acquires, orotherwise obtains one or more image capture correction values, relatedto the exposure time and gain settings, to adjust the image capturesettings of the image acquisition sensor circuitry 110 to minimize,mitigate, adjust, avoid, and/or eliminate the effect of flickeringpixels and/or pixel blocks in the plurality of images. The LFM circuitry120 generates an output signal 202 that includes information and/or datarepresentative of the determined corrected exposure and/or gain valuesto minimize, mitigate, alter, or eliminate the effects of flickeringpixels and/or pixel blocks in the HDR image generated by the HDRstitching circuitry 130.

At 612, the 3A circuitry 210 receives the output signal 202 from the LFMcircuitry 120 and uses the correction factors included in the signal toadjust the exposure and/or gain of at least a portion of the pluralityof images obtained by the image acquisition sensor circuitry 110. Themethod 600 concludes at 614.

FIG. 7 is a high-level flow diagram of an illustrative method 700 ofcorrecting, altering, and/or adjusting the capture settings, such as theexposure time and various gains, of one or more of the images collectedor otherwise acquired by the image acquisition sensor circuitry 110, inaccordance with at least one embodiment described herein. The method 700may be used in conjunction with methods 500 and/or 600, described indetail above with regard to FIGS. 5 and 6. The method 700 commences at702.

At 704, the LFM circuitry 120 determines, calculates, extracts, and/orobtains one or more features associated with each pixel and/or pixelblocks identified as flickering.

At 706, the LFM circuitry 120 compares the determined one or moreextracted features with one or more defined threshold values.

At 708, using the determined one or more extracted features and the oneor more defined threshold values, the LFM circuitry 120 determines oneor more exposure correction values to apply to one or more of theplurality of images generated by the image acquisition sensor circuitry110.

At 710, the LFM circuitry 120 generates one or more output signals 202that include information and/or data representative of the one or moredetermined exposure correction values.

The LFM circuitry 120 communicates the one or more output signals 202 tothe image acquisition sensor circuitry 110. The method 700 concludes at712.

While FIGS. 5, 6, and 7 illustrate various operations according to oneor more embodiments, it is to be understood that not all of theoperations depicted in FIGS. 5, 6, and 7 are necessary for otherembodiments. Indeed, it is fully contemplated herein that in otherembodiments of the present disclosure, the operations depicted in FIGS.5, 6, and 7, and/or other operations described herein, may be combinedin a manner not specifically shown in any of the drawings, but stillfully consistent with the present disclosure. Thus, claims directed tofeatures and/or operations that are not exactly shown in one drawing aredeemed within the scope and content of the present disclosure.

As used in this application and in the claims, a list of items joined bythe term “and/or” can mean any combination of the listed items. Forexample, the phrase “A, B and/or C” can mean A; B; C; A and B; A and C;B and C; or A, B and C. As used in this application and in the claims, alist of items joined by the term “at least one of” can mean anycombination of the listed terms. For example, the phrases “at least oneof A, B or C” can mean A; B; C; A and B; A and C; B and C; or A, B andC.

As used in any embodiment herein, the terms “system” or “module” mayrefer to, for example, software, firmware and/or circuitry configured toperform any of the aforementioned operations. Software may be embodiedas a software package, code, instructions, instruction sets and/or datarecorded on non-transitory computer readable storage mediums. Firmwaremay be embodied as code, instructions or instruction sets and/or datathat are hard-coded (e.g., nonvolatile) in memory devices.

As used in any embodiment herein, the term “circuitry” may comprise, forexample, singly or in any combination, hardwired circuitry, programmablecircuitry such as computer processors comprising one or more individualinstruction processing cores, state machine circuitry, and/or firmwarethat stores instructions executed by programmable circuitry or futurecomputing paradigms including, for example, massive parallelism, analogor quantum computing, hardware embodiments of accelerators such asneural net processors and non-silicon implementations of the above. Thecircuitry may, collectively or individually, be embodied as circuitrythat forms part of a larger system, for example, an integrated circuit(IC), system on-chip (SoC), desktop computers, laptop computers, tabletcomputers, servers, smartphones, etc.

Any of the operations described herein may be implemented in a systemthat includes one or more mediums (e.g., non-transitory storage mediums)having stored therein, individually or in combination, instructions thatwhen executed by one or more processors perform the methods. Here, theprocessor may include, for example, a server CPU, a mobile device CPU,and/or other programmable circuitry. Also, it is intended thatoperations described herein may be distributed across a plurality ofphysical devices, such as processing structures at more than onedifferent physical location. The storage medium may include any type oftangible medium, for example, any type of disk including hard disks,floppy disks, optical disks, compact disk read-only memories (CD-ROMs),compact disk rewritables (CD-RWs), and magneto-optical disks,semiconductor devices such as read-only memories (ROMs), random accessmemories (RAMs) such as dynamic and static RAMs, erasable programmableread-only memories (EPROMs), electrically erasable programmableread-only memories (EEPROMs), flash memories, Solid State Disks (SSDs),embedded multimedia cards (eMMCs), secure digital input/output (SDIO)cards, magnetic or optical cards, or any type of media suitable forstoring electronic instructions. Other embodiments may be implemented assoftware executed by a programmable control device.

Thus, the present disclosure is directed to systems and methods ofdetecting and minimizing the effect of flicker in images obtained usingimage acquisition sensor circuitry. Light flicker mitigation (LFM)circuitry receives a signal containing data representative of aplurality of images. Multi-feature extraction circuits in the LFMcircuitry extract data representative of one or more features from someor all of the pixels included in the plurality of images. Light flickerdetection and analysis circuits in the LFM circuitry detect flickeringpixels in the plurality of images. Exposure index conversion circuits inthe LFM circuitry determine one or more output corrections to apply tothe plurality of images as the images are stitched to form a single HDRimage in which flickering pixels are minimized, mitigated, oreliminated.

The following examples pertain to further embodiments. The followingexamples of the present disclosure may comprise subject material such asat least one device, a method, at least one machine-readable medium forstoring instructions that when executed cause a machine to perform actsbased on the method, means for performing acts based on the methodand/or a system for detecting and minimizing the effect of flicker inmulti-exposure data captures obtained using image acquisition sensorcircuitry.

According to example 1, there is provided an image acquisition system.The system may include: high dynamic range (HDR) sensor circuitry toobtain multi-exposure image data for each of a plurality of pixelsincluded in an image capture; light flicker mitigation (LFM) circuitryto: receive the image data from the image acquisition sensor; performmulti-exposure feature extraction to identify pixels included in theplurality of pixels that demonstrate flickering; determine, on aper-pixel basis, one or more correction factors to minimize, mitigate,reduce, avoid, or eliminate the contribution of pixels identified asdemonstrating flickering in the HDR stitching process; HDR stitchingcircuitry to: adaptively produce, on a per-pixel basis, the displayimage using the image data received from the image acquisition sensorand the one or more correction factors received from the LFM circuitry.

Example 2 may include elements of example 1, and the system mayadditionally include: 3A (Auto-exposure/Auto-focus/Auto White Balance)adjustment circuitry coupled to the LFM circuitry and to the imageacquisition sensor circuitry, the 3A circuitry to: adjust anauto-exposure setting of the image acquisition sensor circuitry usinginformation received from the LFM circuitry indicative of one or moreauto-exposure parameters.

Example 3 may include elements of any of examples 1 or 2 where the LFMcircuitry to further, for each pixel identified as flickering, determine(extract) one or more features associated with the respective pixel;compare the one or more extracted features with one or more definedthreshold values; determine an exposure correction value using the oneor more output values and the one or more defined threshold values; andcommunicate the exposure correction value to the image acquisitionsensor circuitry.

Example 4 may include elements of any of examples 1 through 3 where theone or more extracted features comprise at least one of an averagevalue, a weighted average value, or a maximum value.

Example 5 may include elements of any of examples 1 through 4 where theone or more defined threshold values include at least one of a sensorsaturation value or a scaled or otherwise modified value representativeof a defined sensor saturation value.

Example 6 may include elements of any of examples 1 through 5 where, todetermine an exposure correction value using the one or more extractedfeatures and the one or more defined threshold values, 3A adjustmentcircuitry to further: convert at least one of a difference between theone or more extracted features or a ratio of the one or more extractedfeatures to provide the exposure correction value using the one or moredefined threshold values.

Example 7 may include elements of any of examples 1 through 6 where toperform multi-exposure feature extraction to identify pixels included inthe plurality of pixels that demonstrate flickering, the LFM circuitryto further: perform multi-exposure feature extraction to identify pixelsincluded in the plurality of pixels that demonstrate flickering using atleast one of: a pixel value associated with the respective pixel orluminance components associated with the respective pixel.

Example 8 may include elements of any of examples 1 through 7 where, toperform multi-exposure feature extraction to identify pixels included inthe plurality of pixels that demonstrate flickering, the LFM circuitryto: compare one or more multi-exposure features included in one or morepairs of images obtained by the image acquisition sensor and normalizedfor variations in capture settings for each of the images included ineach of the one or more pairs of images.

Example 9 may include elements of any of examples 1 through 8 where, toperform multi-exposure feature extraction to identify pixels included inthe plurality of pixels that demonstrate flickering, the LFM circuitryto: compare one or more multi-exposure features included in one or morepairs of images obtained by the image acquisition sensor by determininga difference between a first component and a second component using atleast one of: one or more scale factors or one or more offset factors.

Example 10 may include elements of any of examples 1 through 9 where, toperform multi-exposure feature extraction to identify pixels included inthe plurality of pixels that demonstrate flickering, the LFM circuitryto: determine a ratio between at least one of: pixel values associatedwith each respective pixel or luminance components associated with eachrespective pixel to detect flickering.

According to example 11, there is provided an image acquisition method.The method may include: obtaining, via high dynamic range (HDR) sensorcircuitry, multi-exposure image data for each of a plurality of pixelsincluded in an image capture; receiving, by light flicker mitigation(LFM) circuitry, the multi-exposure image data from the imageacquisition sensor; performing, by the LFM circuitry, multi-exposurefeature extraction to identify pixels included in the plurality ofpixels that demonstrate flickering; determining, by the LFM circuitry ona per-pixel basis, one or more correction factors to adjust the outputof pixels identified as demonstrating flickering; and adaptivelyproducing, by HDR stitching circuitry the HDR image using the image datareceived from the image acquisition sensor and the one or morecorrection factors received from the LFM circuitry.

Example 12 may include elements of example 11, and the method mayadditionally include: adjusting, by 3A (Auto-exposure/Auto-focus/AutoWhite Balance) adjustment circuitry coupled to the LFM circuitry and tothe image acquisition sensor circuitry, an auto-exposure setting of theimage acquisition sensor circuitry using information received from theLFM circuitry indicative of one or more auto exposure parameters.

Example 13 may include elements of any of examples 11 or 12, and themethod may further include: determining, by the LFM circuitry, for oneor more extracted features associated with each pixel identified asflickering; comparing, by the 3A adjustment circuitry, the one or moreextracted features with one or more defined threshold values;determining, by the 3A adjustment circuitry, an exposure correctionvalue using the one or more extracted features and the one or moredefined threshold values; and communicating, by the 3A adjustmentcircuitry, the exposure correction value to the image acquisition sensorcircuitry.

Example 14 may include elements of any of examples 11 through 13 wherecomparing the one or more extracted features with the one or moredefined threshold values comprises: comparing, by the 3A adjustmentcircuitry, at least one of an average value, a weighted average value,or a maximum value with the one or more defined threshold values.

Example 15 may include elements of any of examples 11 through 14 wheredetermining an exposure correction value using the one or more extractedfeatures and the one or more defined threshold values comprises: using,by the 3A adjustment circuitry, at least one of a sensor saturationvalue or a scaled or otherwise modified value representative of adefined sensor saturation value as the one or more defined thresholdvalues.

Example 16 may include elements of any of examples 11 through 15 wheredetermining an exposure correction value using the one or more extractedfeatures and the one or more defined threshold values comprises:converting, by the 3A adjustment circuitry, at least one of a differencebetween the one or more extracted features or a ratio of the one or moreextracted features to provide the exposure correction value using theone or more defined threshold values.

Example 17 may include elements of any of examples 11 through 16 whereperforming multi-exposure feature extraction to identify pixels includedin the plurality of pixels that demonstrate flickering furthercomprises: performing, by the LFM circuitry, multi-exposure featureextraction to identify pixels included in the plurality of pixels thatdemonstrate flickering using at least one of a pixel value associatedwith the respective pixel or luminance components associated with therespective pixel.

Example 18 may include elements of any of examples 11 through 17 whereperforming multi-exposure feature extraction to identify pixels includedin the plurality of pixels that demonstrate flickering furthercomprises: comparing, by the LFM circuitry, one or more multi-exposurefeatures included in one or more pairs of images obtained by the imageacquisition sensor and normalized for variations in capture settings foreach of the images included in each of the one or more pairs of images.

Example 19 may include elements of any of examples 11 through 18 whereperforming multi-exposure feature extraction to identify pixels includedin the plurality of pixels that demonstrate flickering furthercomprises: comparing, by the LFM circuitry, one or more multi-exposurefeatures included in one or more pairs of images obtained by the imageacquisition sensor by determining a difference between a first componentand a second component using at least one of one or more scale factorsor one or more offset factors.

Example 20 may include elements of any of examples 11 through 19 whereperforming multi-exposure feature extraction to identify pixels includedin the plurality of pixels that demonstrate flickering furthercomprises: determining, by the LFM circuitry, a ratio between at leastone of pixel values associated with each respective pixel or luminancecomponents associated with each respective pixel to detect flickering.

According to example 21, there is provided a non-transitory storagedevice. The non-transitory storage device includes instructions that,when executed by light flicker mitigation (LFM) circuitry, cause the LFMcircuitry to: receive, from high dynamic range (HDR) sensor circuitry,multi-exposure image data for each of a plurality of pixels included inan image capture; perform multi-exposure feature extraction to identifypixels included in the plurality of pixels that demonstrate flickering;determine, on a per-pixel basis, one or more correction factors toadjust the output of pixels identified as demonstrating flickering; andcause HDR stitching circuitry to adaptively produce, on a per-pixelbasis, the display image using the image data received from the imageacquisition sensor and the one or more correction factors received fromthe LFM circuitry.

Example 22 may include elements of example 21 where the instructionsfurther cause the LFM circuitry to: cause 3A(Auto-exposure/Auto-focus/Auto White Balance) adjustment circuitry toadjust an auto-exposure setting of the image acquisition sensorcircuitry using information received from the LFM circuitry indicativeof one or more auto exposure parameters.

Example 23 may include elements of any of examples 21 or 22 where theinstructions further cause the LFM circuitry to: determine (extract) oneor more features associated with each pixel identified as flickering;cause the 3A adjustment circuitry to compare the one or more extractedfeatures with one or more defined threshold values; cause the 3Aadjustment circuitry to determine an exposure correction value using theone or more extracted features and the one or more defined thresholdvalues; and cause the 3A adjustment circuitry to communicate theexposure correction value to the image acquisition sensor circuitry.

Example 24 may include elements of any of examples 21 through 23 wherethe instructions that cause the LFM circuitry to cause the 3A adjustmentcircuitry to compare the one or more extracted features with one or moredefined threshold values further cause the LFM circuitry to: cause the3A adjustment circuitry to compare at least one of an average value, aweighted average value, or a maximum value with the one or more definedthreshold values. Example 25 may include elements of any of examples 21through 24 where the instructions that cause the LFM circuitry to causethe 3A adjustment circuitry to determine an exposure correction valueusing the one or more extracted features and the one or more definedthreshold values further cause the LFM circuitry to: cause the 3Aadjustment circuitry to determine the exposure correction value usingone or more defined threshold values that includes at least one of asensor saturation value or a scaled value representative of a definedsensor saturation value.

Example 26 may include elements of any of examples 21 through 25 wherethe instructions that cause the LFM circuitry to cause the 3A adjustmentcircuitry to determine an exposure correction value using the one ormore extracted features and the one or more defined threshold valuesfurther cause the LFM circuitry to: cause the 3A adjustment circuitry toconvert to the one or more defined threshold values to provide theexposure correction value using the one or more defined threshold valuesand at least one of a difference between the one or more extractedfeatures and or a ratio of the one or more extracted features.

Example 27 may include elements of any of examples 21 through 26 wherethe instructions that cause the LFM circuitry to perform themulti-exposure feature extraction to identify pixels included in theplurality of pixels that demonstrate flickering further cause the LFMcircuitry to: perform the multi-exposure feature extraction to identifypixels included in the plurality of pixels that demonstrate flickeringusing at least one of a pixel value associated with the respective pixelor one or more luminance components associated with the respectivepixel.

Example 28 may include elements of any of examples 21 through 27 wherethe instructions that cause the LFM circuitry to perform themulti-exposure feature extraction to identify pixels included in theplurality of pixels that demonstrate flickering further cause the LFMcircuitry to: compare one or more multi-exposure features included inone or more pairs of images obtained by the image acquisition sensor andnormalized for variations in capture settings for each of the imagesincluded in each of the one or more pairs of images.

Example 29 may include elements of any of examples 21 through 28 wherethe instructions that cause the LFM circuitry to perform themulti-exposure feature extraction to identify pixels included in theplurality of pixels that demonstrate flickering further cause the LFMcircuitry to: compare the one or more multi-exposure features includedin one or more pairs of images obtained by the image acquisition sensorby determining a difference between a first component and a secondcomponent using at least one of one or more scale factors or one or moreoffset factors.

Example 30 may include elements of any of examples 21 through 29 wherethe instructions that cause the LFM circuitry to perform themulti-exposure feature extraction to identify pixels included in theplurality of pixels that demonstrate flickering further cause the LFMcircuitry to: determine a ratio between at least one of pixel valuesassociated with each respective pixel or luminance components associatedwith each respective pixel to detect flickering.

According to example 31, there is provided an image acquisition system.The system may include: means for obtaining, multi-exposure image datafor each of a plurality of pixels included in an image; means forreceiving the multi-exposure image data from the image acquisitionsensor; means for performing multi-exposure feature extraction toidentify pixels included in the plurality of pixels that demonstrateflickering; means for determining, on a per-pixel basis, one or morecorrection factors to minimize, mitigate, avoid, reduce, or eliminatethe contributions of pixels identified as demonstrating flickering; andmeans for adaptively producing, on a per-pixel basis, the display imageusing the image data received from the image acquisition sensor and theone or more correction factors received from the light flickermitigation circuitry.

Example 32 may include elements of example 31, and the system mayfurther include: means for adjusting an auto-exposure setting of theimage acquisition sensor circuitry using information received from theLFM circuitry indicative of one or more auto exposure parameters.

Example 33 may include elements of any of examples 31 or 32, and thesystem may further include: means for determining (extracting) one ormore features associated with each pixel identified as flickering; meansfor comparing the one or more extracted features with one or moredefined threshold values; means for determining an exposure correctionvalue using the one or more extracted features and the one or moredefined threshold values; and means for communicating the exposurecorrection value to the image acquisition sensor circuitry.

Example 34 may include elements of any of examples 31 through 33 wherethe means for comparing the one or more extracted features with the oneor more defined threshold values comprises: means for comparing, by the3A adjustment circuitry, at least one of an average value, a weightedaverage value, or a maximum value with the one or more defined thresholdvalues.

Example 35 may include elements of any of examples 31 through 34 wherethe means for determining the exposure correction value using the one ormore extracted features and the one or more defined threshold valuescomprises: means for obtaining at least one of a sensor saturation valueor a scaled value representative of a defined sensor saturation value.

Example 36 may include elements of any of examples 31 through 35 wherethe means for determining the exposure correction value using the one ormore extracted features and the one or more defined threshold valuescomprises: means for converting at least one of a difference between theone or more extracted features or a ratio of the one or more extractedfeatures using the one or more defined threshold values to provide theexposure correction value.

Example 37 may include elements of any of examples 31 through 36 wherethe means for performing the multi-exposure feature extraction toidentify pixels included in the plurality of pixels that demonstrateflickering further comprises: using at least one of a pixel valueassociated with the respective pixel or one or more luminance componentsassociated with the respective pixel.

Example 38 may include elements of any of examples 31 through 37 wherethe means for performing the multi-exposure feature extraction toidentify pixels included in the plurality of pixels that demonstrateflickering further comprises: means for comparing one or moremulti-exposure features included in one or more pairs of imagesnormalized for variations in capture settings for each of the imagesincluded in each of the one or more pairs of images.

Example 39 may include elements of any of examples 31 through 38 wherethe means for performing the multi-exposure feature extraction toidentify pixels included in the plurality of pixels that demonstrateflickering further comprises: means for comparing one or moremulti-exposure features included in one or more pairs of images obtainedby the image acquisition sensor by determining a difference between afirst component and a second component using at least one of one or morescale factors or one or more offset factors.

Example 40 may include elements of any of examples 31 through 39 wherethe means for performing the multi-exposure feature extraction toidentify pixels included in the plurality of pixels that demonstrateflickering further comprises: means for determining a ratio between atleast one of pixel values associated with each respective pixel orluminance components associated with each respective pixel to detectflickering.

According to example 41, there is provided a system for provision ofdetecting and minimizing the effect of flicker in images obtained usingimage acquisition sensor circuitry, the system being arranged to performthe method of any of examples 11 through 20.

According to example 42, there is provided a chipset arranged to performthe method of any of examples 11 through 20.

According to example 43, there is provided at least one machine-readablestorage device that includes a plurality of instructions that, inresponse to be being executed on a computing device, cause the computingdevice to carry out the method according to any of examples 11 through20.

According to example 44, there is provided a device configured fordetecting and minimizing the effect of flicker in images obtained usingimage acquisition sensor circuitry, the device being arranged to performthe method of any of the examples 11 through 20.

The terms and expressions which have been employed herein are used asterms of description and not of limitation, and there is no intention,in the use of such terms and expressions, of excluding any equivalentsof the features shown and described (or portions thereof), and it isrecognized that various modifications are possible within the scope ofthe claims. Accordingly, the claims are intended to cover all suchequivalents. Various features, aspects, and embodiments have beendescribed herein. The features, aspects, and embodiments are susceptibleto combination with one another as well as to variation andmodification, as will be understood by those having skill in the art.The present disclosure should, therefore, be considered to encompasssuch combinations, variations, and modifications.

As described herein, various embodiments may be implemented usinghardware elements, software elements, or any combination thereof.Examples of hardware elements may include processors, microprocessors,circuits, circuit elements (e.g., transistors, resistors, capacitors,inductors, and so forth), integrated circuits, application specificintegrated circuits (ASIC), programmable logic devices (PLD), digitalsignal processors (DSP), field programmable gate array (FPGA), logicgates, registers, semiconductor device, chips, microchips, chip sets,and so forth.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. Thus, appearances of the phrases “in oneembodiment” or “in an embodiment” in various places throughout thisspecification are not necessarily all referring to the same embodiment.Furthermore, the particular features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments.

What is claimed:
 1. An image acquisition system, comprising: highdynamic range (HDR) sensor circuitry configured to obtain multi-exposureimage data for each of a plurality of pixels included in an image; lightflicker mitigation (LFM) circuitry configured to: receive themulti-exposure image data from the an image acquisition sensorcircuitry; perform multi-exposure feature extraction to identify pixelsincluded in the plurality of pixels that demonstrate flickering; convertthe identified pixels that demonstrate flickering to one or more valuesrepresentative of a first exposure index that indicates a contributionof the identified pixels that demonstrate flickering to a correctedpixel in a stitched HDR image by assigning, to individual exposureimages from among the multi-exposure image data, values within adetection range associated with the HDR sensor circuitry; repeat theconversion of identified pixels that demonstrate flickering for pairs ofimages from among the multi-exposure image data to calculate a secondexposure index; determine one or more correction parameters based uponthe second exposure index to reduce, in the stitched HDR image, aflickering effect of the pixels identified as demonstrating flickering;and HDR stitching circuitry to selectively generate the stitched HDRimage using (i) the multi-exposure image data received from the imageacquisition sensor circuitry, and (ii) the one or more correctionparameters received from the LFM circuitry.
 2. The system of claim 1,further comprising: auto-focus, auto-exposure, auto-white balance (3A)circuitry coupled to the LFM circuitry and to the image acquisitionsensor circuitry, the 3A circuitry being configured to adjust anauto-exposure setting of the image acquisition sensor circuitry usinginformation received from the LFM circuitry indicative of one or moreexposure correction parameters.
 3. The system of claim 2, wherein theLFM circuitry is further configured to: for each pixel identified asflickering, determine one or more features associated with therespective pixel; compare the one or more extracted features with one ormore defined threshold values; determine the one or more exposurecorrection parameters using the one or more extracted features and theone or more defined threshold values; and communicate the one or moreexposure correction parameters to the image acquisition sensorcircuitry.
 4. The system of claim 3, wherein the one or more extractedfeatures comprise at least one of an average value, a weighted averagevalue, a luminance value, or a maximum value.
 5. The system of claim 3,wherein the one or more defined threshold values include at least one ofa sensor saturation value, a scaled value representative of a definedsensor saturation value, or a user-defined value.
 6. The system of claim3, wherein the 3A circuitry is further configured to convert at leastone of (i) a difference between the one or more extracted features, or(ii) a ratio of the one or more extracted features, using the one ormore defined threshold values, to determine the one or more exposurecorrection parameters.
 7. The system of claim 1, wherein the LFMcircuitry is further configured to perform the multi-exposure featureextraction to identify pixels included in the plurality of pixels thatdemonstrate flickering by: identifying pixels included in the pluralityof pixels that demonstrate flickering using at least one of a pixelvalue associated with the respective pixel, luminance componentsassociated with the respective pixel, or other representative valuesassociated with the respective pixel; and performing enhancement, for atleast one of extracted features or obtained correction parameters, usingat least one of low-pass filtering and morphological processing.
 8. Thesystem of claim 7, wherein the LFM circuitry is further configured toperform multi-exposure feature extraction to identify pixels included inthe plurality of pixels that demonstrate flickering by comparing one ormore multi-exposure features included in one or more pairs of imagesobtained by the image acquisition sensor circuitry and normalized forvariations in capture settings, for each of the images included in eachof the one or more pairs of images.
 9. The system of claim 7, whereinthe LFM circuitry is further configured to perform multi-exposurefeature extraction to identify pixels included in the plurality ofpixels that demonstrate flickering by comparing one or moremulti-exposure features included in one or more pairs of images obtainedby the image acquisition sensor circuitry by determining a differencebetween a first component and a second component using at least one ofone or more scale factors or one or more offset factors.
 10. The systemof claim 7, wherein the LFM circuitry is further configured to performmulti-exposure feature extraction to identify pixels included in theplurality of pixels that demonstrate flickering by: determining a ratiobetween at least one of pixel values associated with each respectivepixel, luminance components, or other representative values associatedwith each respective pixel to detect flickering; and comparing thedetermined ratio with one or more threshold values.
 11. The system ofclaim 1, wherein the LFM circuitry is configured to assign, to theindividual exposure images from among the multi-exposure image data,values within the detection range based upon different exposure times ofthe individual exposure images.
 12. A non-transitory storage device thatincludes instructions that, when executed by light flicker mitigation(LFM) circuitry, cause the LFM circuitry to: receive, from imageacquisition sensor circuitry, image data for each of a plurality ofpixels included in each of a plurality of images; perform multi-exposurefeature extraction to identify pixels included in the plurality ofimages that demonstrate flickering; convert the identified pixels thatdemonstrate flickering to one or more values representative of a firstexposure index that indicates a contribution of the identified pixelsthat demonstrate flickering to a corrected pixel in a stitched highdynamic range (HDR) image by assigning, to individual exposure imagesfrom among the multi-exposure image data, values within a detectionrange associated with a HDR sensor circuitry; repeat the conversion ofidentified pixels that demonstrate flickering for pairs of images fromamong the multi-exposure image data to calculate a second exposureindex; determine one or more correction parameters based upon the secondexposure index to reduce the contributions of pixels identified asdemonstrating flickering to an output pixel; and cause HDR stitchingcircuitry to selectively and adaptively combine the plurality of imagesusing the image data received from the image acquisition sensorcircuitry and the one or more correction parameters received from theLFM circuitry to generate the stitched HDR image.
 13. The non-transitorystorage device of claim 12 wherein the instructions further cause theLFM circuitry to: cause auto-focus, auto-exposure, auto-white balance(3A) circuitry to adjust an auto-exposure setting of the imageacquisition sensor circuitry using information received from the LFMcircuitry indicative of one or more exposure correction parameters. 14.The non-transitory storage device of claim 13, wherein the instructionsfurther cause the LFM circuitry to: determine one or more featuresassociated with each pixel identified as flickering; cause the 3Aadjustment circuitry to compare the one or more extracted features withone or more defined threshold values; cause the 3A adjustment circuitryto determine the one or more exposure correction parameters using theone or more extracted features and the one or more defined thresholdvalues; and cause the 3A adjustment circuitry to communicate the one ormore exposure correction parameters to the image acquisition sensorcircuitry.
 15. The non-transitory storage device of claim 14, whereinthe instructions that cause the LFM circuitry to cause the 3A adjustmentcircuitry to compare the one or more extracted features with one or moredefined threshold values further cause the LFM circuitry to: cause the3A adjustment circuitry to compare the one or more extracted featuresincluding at least one of an average value, a weighted average value,luminance value, or a maximum value with the one or more definedthreshold values.
 16. The non-transitory storage device of claim 14,wherein the instructions that cause the LFM circuitry to cause the 3Aadjustment circuitry to determine the one or more exposure correctionparameters using the one or more extracted features and the one or moredefined threshold values further cause the LFM circuitry to: cause the3A adjustment circuitry to determine the one or more exposure correctionparameters using one or more defined threshold values that include atleast one of a sensor saturation value, a scaled value representative ofa defined sensor saturation value, or a user-defined value.
 17. Thenon-transitory storage device of claim 14, wherein the instructions thatcause the LFM circuitry to cause the 3A circuitry to determine the oneor more exposure correction parameters using the one or more extractedfeatures and the one or more defined threshold values further cause theLFM circuitry to: cause the 3A circuitry to provide the one or moreexposure correction parameters using the one or more threshold valuesand at least one of a difference between the one or more extractedfeatures or a ratio of the one or more extracted features.
 18. Thenon-transitory storage device of claim 12, wherein the instructions thatcause the LFM circuitry to perform the multi-exposure feature extractionto identify pixels included in the plurality of pixels that demonstrateflickering further cause the LFM circuitry to: perform themulti-exposure feature extraction to identify pixels included in theplurality of pixels that demonstrate flickering using at least one of apixel value associated with the respective pixel, one or more luminancecomponents associated with the respective pixel, or one or more otherrepresentative values associated with the respective pixels; and performenhancement, for at least one of extracted features or obtainedcorrection parameters, using at least one of low-pass filtering andmorphological processing.
 19. The non-transitory storage device of claim18, wherein the instructions that cause the LFM circuitry to perform themulti-exposure feature extraction to identify pixels included in theplurality of pixels that demonstrate flickering further cause the LFMcircuitry to: compare one or more multi-exposure features included inone or more pairs of images obtained by the image acquisition sensorcircuitry and normalized for variations in capture settings, for each ofthe images included in each of the one or more pairs of images.
 20. Thenon-transitory storage device of claim 18, wherein the instructions thatcause the LFM circuitry to perform the multi-exposure feature extractionto identify pixels included in the plurality of pixels that demonstrateflickering further cause the LFM circuitry to: compare the one or moremulti-exposure features included in one or more pairs of images obtainedby the image acquisition sensor circuitry by determining a differencebetween a first component and a second component using at least one ofone or more scale factors or one or more offset factors.
 21. Thenon-transitory storage device of claim 18, wherein the instructions thatcause the LFM circuitry to perform the multi-exposure feature extractionto identify pixels included in the plurality of pixels that demonstrateflickering further cause the LFM circuitry to: determine a ratio betweenat least one of pixel values associated with each respective pixel,luminance components associated with each respective pixel, or otherrepresentative values associated with each respective pixel to detectflickering; and compare the determined ratio with one or more thresholdvalues.
 22. An image acquisition system, comprising: means for obtainingimage data for each of a plurality of pixels included in each of aplurality of images; means for receiving the obtained image data from animage acquisition sensor circuitry; means for performing multi-exposurefeature extraction to identify pixels included in the plurality ofimages that demonstrate flickering; means for converting the identifiedpixels that demonstrate flickering to one or more values representativeof a first exposure index that indicates a contribution of theidentified pixels that demonstrate flickering to a corrected pixel in astitched high dynamic range (HDR) image by assigning, to individualexposure images from among multi-exposure image data, values within adetection range associated with a HDR sensor circuitry; means forrepeating the conversion of identified pixels that demonstrateflickering for pairs of images from among the multi-exposure image datato calculate a second exposure index; means for determining one or morecorrection parameters based upon the second exposure index to adjust thecontributions of the pixels identified as demonstrating flickering to anoutput pixel; and means for selectively stitching the plurality ofimages using the image data received from the image acquisition sensorcircuitry and the one or more correction parameters received from lightflicker mitigation (LFM) circuitry to generate the stitched HDR image.23. The system of claim 22, further comprising: means for adjusting anauto-exposure setting of the image acquisition sensor circuitry usinginformation received from the LFM circuitry indicative of the one ormore exposure correction parameters.
 24. The system of claim 23, furthercomprising: means for determining one or more features associated witheach pixel identified as flickering; means for comparing the one or moreextracted features with one or more defined threshold values; means fordetermining the one or more exposure correction parameters using the oneor more extracted features and the one or more defined threshold values;and means for communicating the one or more exposure correctionparameters to the image acquisition sensor circuitry.
 25. The system ofclaim 24, wherein the means for comparing the one or more extractedfeatures with the one or more defined threshold values comprises: meansfor comparing, by auto-focus, auto-exposure, auto-white balance (3A)circuitry circuitry, at least one of an average value, a weightedaverage value, a luminance value, or a maximum value as the one or moreextracted features with the one or more defined threshold values. 26.The system of claim 24, wherein the means for determining the one ormore exposure correction parameters using the one or more extractedfeatures and the one or more defined threshold values comprises: meansfor determining the one or more exposure correction parameters using theone or more extracted features and the one or more defined thresholdvalues including at least one of a sensor saturation value, a scaledvalue representative of a defined sensor saturation value, or auser-defined value.
 27. The system of claim 24, wherein the means fordetermining the one or more exposure correction parameters using the oneor more extracted features and the one or more defined threshold valuescomprises: means for converting at least one of a difference between theone or more extracted features or a ratio of the one or more extractedfeatures using the one or more defined threshold values to provide theone or more exposure correction parameters.
 28. The system of claim 22,wherein the means for performing the multi-exposure feature extractionto identify pixels included in the plurality of pixels that demonstrateflickering further comprises: means for performing the multi-exposurefeature extraction to identify pixels included in the plurality ofpixels that demonstrate flickering using at least one of a pixel valueassociated with the respective pixel, one or more luminance componentsassociated with the respective pixel, or one or more otherrepresentative values associated with the respective pixel; and meansfor performing enhancement, for at least one of extracted features orobtained correction parameters, using at least one of low-pass filteringand morphological processing.
 29. The system of claim 28, wherein themeans for performing the multi-exposure feature extraction to identifypixels included in the plurality of pixels that demonstrate flickeringfurther comprises: means for comparing one or more multi-exposurefeatures included in one or more pairs of images normalized forvariations in capture settings for each of the images included in eachof the one or more pairs of images.
 30. The system of claim 28, whereinthe means for performing the multi-exposure feature extraction toidentify pixels included in the plurality of pixels that demonstrateflickering further comprises: means for comparing one or moremulti-exposure features included in one or more pairs of images obtainedby the image acquisition sensor circuitry by determining a differencebetween a first component and a second component using at least one ofone or more scale factors or one or more offset factors.
 31. The systemof claim 28, wherein the means for performing the multi-exposure featureextraction to identify pixels included in the plurality of pixels thatdemonstrate flickering further comprises: means for determining a ratiobetween at least one of pixel values associated with each respectivepixel, luminance components associated with each respective pixel, orother representative values associated with each respective pixel todetect flickering; means for comparing the determined ratio with one ormore threshold values.